Shadow in an image is important visual information of the projective object, but it affects computer vision tasks. Existing single image shadow removal methods cannot obtain good shadow-free results due to the lack of robust shadow features or insufficiency of and errors in training sample data. In order to generate accurately the shadow mask image for describing the illumination attenuation degree and obtain the high quality shadow-free image, a single image shadow removal method based on attenuated generative adversarial network was proposed. Firstly, an attenuator guided by the sensitive parameters was used to augment the training sample data in order to provide shadow sample images agreed with physical illumination model for a subsequent generator and discriminator. Then, with the supervision from the discriminator, the generator combined perceptual loss function to generate the final shadow mask. Compared with related works, the proposed method can effectively recover the illumination information of shadow regions and obtain the more realistic shadow-free image with natural transition of shadow boundary. Shadow removal results were evaluated using objective metric. Experimental results show that the proposed method can remove shadow effectively in various real scenes with a good visual consistency.

In order to solve spatio-temporally incoherent problem of traditional shadow removal methods for videos captured by free moving cameras, a shadow detection and removal approach using region matching guided by illumination transfer was proposed. Firstly, the input video was segmented by using Mean Shift method based on Scale Invariant Feature Transform (SIFT), and the video shadow was detected by Support Vector Machine (SVM) classifier. Secondly, the input video was decomposed into overlapped 2D patches, and a Markov Random Field (MRF) for this video was set up, and the corresponding lit patch for every shadow patch was found via region matching guided by optical flow. Finally, in order to get spatio-temporally coherent results, each shadow patch was processed with its matched lit patch by local illumination transfer operation and global shadow removal. The experimental results show that the proposed algorithm obtains higher accuracy and lower error than the traditional methods based on illumination transfer, the comprehensive evaluation metric is improved by about 6.23%, and the Root Mean Square Error (RMSE) is reduced by about 30.12%. It can obtain better shadow removal results with more spatio-temporal coherence but much less time.

Focusing on the issues of poor depth estimation and inaccurate depth value acquisition under traditional machine learning methods, a depth estimation model based on Atrous Convolutional Neural Network (ACNN) was proposed. Firstly, the feature map of original image was extracted layer by layer using Convolutional Neural Network (CNN). Secondly, with the atrous convolution structure, the spatial information in original image and the extracted feature map were fused to obtain initial depth map. Finally, the Conditional Random Field (CRF) with combining three constraints, pixel spatial position, grayscale and gradient information were used to optimize initial depth map and obtain final depth map. The model usability verification and error estimation were completed on objective data set. The experimental results show that the proposed algorithm obtains lower error value and higher accuracy. The Root Mean Square Error (RMS) is averagely reduced by 30.86% compared with machine learning based algorithm, and the accuracy is improved by 14.5% compared with deep learning based algorithm. The proposed algorithm has a significant improvement in error reduction and visual effect, indicating that the model can obtain better results in image depth estimation.

Under MapReduce computing scenarios, complex data mining algorithms typically require multiple MapReduce jobs' collaboration process to compete the task. However, serious redundant disk read and write and repeat resource request operations among multiple MapReduce jobs seriously degrade the performance of the algorithm under MapReduce. To improve the computational efficiency of ItemBased recommendation algorithm, firstly, the performance issues of the ItemBased collaborative filtering algorithm under MapReduce platform were analyzed. Secondly, the execution efficiency of the algorithm was improved by taking advantage of Spark's performance superiority on iterative computation and memory computing, and the ItemBased collaborative filtering algorithm under Spark platform was implemented. The experimental results show that, when the size of the cluster nodes is 10 and 20, the running time of the algorithm in Spark is only 25.6% and 30.8% of that in MapReduce. The algorithm's overall computing efficiency of Spark platform improves more than 3 times compared with that of MapReduce platform.

In the field of big data real-time analysis and computing, the importance of stream computing is constantly improved while the energy consumption of dealing with data on stream computing platform rises constantly. In order to solve the problems, an Energy-efficient Strategy for Threshold Control (ESTC) was proposed by changing the processing mode of node to data in stream computing. First of all, according to system load difference, the threshold of the work node was determined. Secondly, according to the threshold of the work node, the system data stream was randomly selected to determine the physical voltage of the adjustment system in different data processing situation. Finally, the system power was determined according to the different physical voltage. The experimental results and theoretical analysis show that in stream computing cluster consisting of 20 normal PCs, the system based on ESTC saves about 35.2% more energy than the original system. In addition, the ratio of performance and energy consumption under ESTC is 0.0803 tuple/(s·J), while the original system is 0.0698 tuple/(s·J). Therefore, the proposed ESTC can effectively reduce the energy consumption without affecting the system performance.

Concerning the problem of data sparseness, cold start and low user experience of recommendation system, a video recommendation algorithm based on clustering and hierarchical model was proposed to improve the performance of recommendation system and user experience. Focusing on the user, similar users were obtained by analyzing Affiliation Propagation (AP) cluster, then historical data of online video of similar users was collected and a recommendation set of videos was geberated. Secondly, the user preference degree of a video was calculated and mapped into the tag weight of the video. Finally, a recommendation list of videos was generated by using analytic hierarchy model to calculate the ranking of user preference with videos. The experimental results on MovieLens Latest Dataset and YouTube video review text dataset show that the proposed algorithm has good performance in terms of Root-Mean-Square Error (RMSE) and the recommendation accuracy.

The fairness of multi-resource allocation is one of the most important indicators in the resource scheduling subsystem, Dominant Resource Fairness (DRF), as a general resource allocation algorithm for multi-resources scenarios, it may be unfair in heterogeneous cluster environment. On the basis of the research on the DRF multi-resource fair allocation algorithm under Mesos framework environment, meDRF allocation algorithm was designed and implemented to evaluate the influence factors of the performance of the server. The machine performance scores of computing nodes, as the dominant factor of DRF share calculation, made computing tasks have equal chance to obtain high quality computing resources and poor computing resources. Experiments were conducted by using K-means, Bayes and PageRank jobs under Hadoop. The experimental results show that, compared with DRF allocation algorithm, the meDRF algorithm can reflect more fairness of the allocation of resources, and the allocation of resources has better stability, which effectively improves the utilization of system resources.

In order to build energy-saving green database, a database model of dynamic power consumption based on the smallest unit of Structured Query Language (SQL) resource (Central Processing Unit (CPU), disk) consumption. The proposed model profiled the dynamic power consumption and mapped the main hardwares (CPU, disk) resource consumption to power consumption. Key parameters of the model were fitted by adopting the method of multiple linear regression to estimate the dynamic system power in real-time and build the unit-unified model of dynamic power consumption. The experimental results show that, compared with the model based on the total number of tuples, the total number of CPU instructions can better reflect the CPU power consumption. The average relative error of the constitutive model is less than 6% and the absolute error of the constitutive model is less than 9% while the DataBase Management System (DBMS) monopolizes system resources in the static environment. The proposed dynamic power consumption model is more suitable for building energy-saving green database.

Constant expansion and that energy consumption factors are ignored with its design process, bring the problem of high energy consumption and low efficiency of the cloud storage system. And this problem has become a main bottleneck in the development of cloud computing and big data. Most of previous studies had been mostly used to adjust the entire storage node to the low-power mode to save energy. According to the repetition of data and access rules, new storage model based on data classification was proposed. The storage area was divided into HotZone, ColdZone and ReduplicationZone so as to divisionally store the data according to the repetition and activity factor characteristics of each data file. Based on the new storage model, an energy-efficient storage algorithm was designed and a new storage model was constructed. The experimental results show that, the new storage model improves the energy utilization rate of the distributed storage system nearly 25%, especially when the system load is lower than the given threshold.

Like MapReduce, tasks under big data environment are always with data-dependent constraints. The resource selection strategy in distributed storage system trends to choose the nearest data block to requestor, which ignored the server's resource load state, like CPU, disk I/O and network, etc. On the basis of the distributed storage system's cluster structure, data file division mechanism and data block storage mechanism, this paper defined the cluster-node matrix, CPU load matrix, disk I/O load matrix, network load matrix, file-division-block matrix, data block storage matrix and data block storage matrix of node status. These matrixes modeled the relationship between task and its data constraints. And the article proposed an optimal resource selection algorithm with data-dependent constraints (ORS2DC), in which the task scheduling node is responsible for base data maintenance, MapRedcue tasks and data block read tasks take different selection strategies with different resource-constraints. The experimental results show that, the proposed algorithm can choose higher quality resources for the task, improve the task completion quality while reducing the NameNode's load burden, which can reduce the probability of the single point of failure.

Concerning the shopping information Web page constructed by template, and the large number of Web information and complex Web structure, this paper studied how to extract the shopping information from the Web page template by not using the complex learning rule. The paper defined the Web page template and the extraction template of Web page and designed template language that was used to construct the template. This paper also gave a model of extraction based on template. The experimental results show that the recall rate of the proposed method is 12% higher than the Extraction problem Algorithm (EXALG) by testing the standard 450 Web pages; the results also show that the recall rate of this method is 7.4% higher than Visual information and Tag structure based wrapper generator (ViNTs) method and 0.2% higher than Augmenting automatic information extraction with visual perceptions (ViPER) method and the accuracy rate of this method is 5.2% higher than ViNTs method and 0.2% higher than ViPER method by testing the standard 250 Web pages. The recall rate and the accuracy rate of the extraction method based on the rapid construction template are improved a lot which makes the accuracy of the Web page analysis and the recall rate of the information in the shopping information retrieval and the shopping comparison system improve a lot .

Concerning the poor quality of recommendations of traditional user-based and item-based collaborative filtering models, a new collaborative filtering model combining users and items predictions was proposed. Firstly, it considered both users and items, and optimized the similarity model with excellent performance dynamically. Secondly, it constructed neighbor sets for the target objects by selecting some similar users and items according to the similarity values, and then obtained the user-based and item-based prediction results respectively based on some prediction functions. Finally, it gained final predictions by using the adaptive balance factor to coordinate both of the prediction results. Comparative experiments were carried out under different evaluation criteria, and the results show that, compared with some typical collaborative filtering models such as RSCF, HCFR and UNCF, the proposed model not only has better performance in prediction accuracy of items, but also does well in the precision and recall of recommendations.

An adaptive Virtual Machine (VM) dynamic migration strategy of soft energy-saving was put forward to optimize energy consumption and load balance in cloud computing. The energy-saving strategy adopted Dynamic Voltage Frequency Scaling (DVFS) as the static energy-aware technology to achieve the sub-optimized static energy saving, and used online VM migration to achieve an adaptive dynamic soft energy-saving in cloud platform. The two energy-saving strategies were simulated and compared with each other in CloudSim platform, and the data were tested on PlanetLab platform. The results show that: Firstly, the adaptive soft and hard combination strategy in energy-saving can significantly save 96% energy; secondly, DVFS+MAD_MMT strategy using Median Absolute Deviation (MAD) to determine whether the host is overload, and choosing VM to remove based on Minimum Migration Time (MMT), which can save energy about 87.15% with low-load in PlanetLab Cloudlets than that of experimental environment; finally, security threshold of 2.5 in MAD_MMT algorithm can consume the energy efficiently and achieve the adaptive load balancing of virtual machines migration dynamically.

The data block storage mechanism and downloading process in Hadoop Distributed File System (HDFS) cluster were analyzed. In combination with multi-point and multi-threaded Peer-to-Peer (P2P) download idea, an efficiency optimization algorithm was proposed from the aspects of data-block, file and cluster. Concerning the possible imbalanced load problem caused by multi-thread download in HDFS cluster, a download-point selection algorithm was put forward to optimize the download-point selection. The mathematical analysis and experiments prove that the three methods can improve the download efficiency and download-point selection algorithm can achieve loading balance among DataNodes in HDFS cluster.

With the development of MIPv6 and wireless technology, the AAA Architecture of WLAN based on MIPv6 has attracted more and more attention. Because the foreign Agent does not exist in IPv6, the access router can be attendant of the AAA architecture based on Diameter. It can not only control the mobile node access the WLAN, but can be client of the Diameter AAA architecture. The netfilter frame in the Linux Kernel 2.4 has the powerful function of processing packet, and it can be used in the access router implementation.